Week 10 of 12 · The Lead

Onboarding & the Throughput–Quality Report

Two things a lead owns that no one else will: how a new annotator earns the live queue, and how you tell leadership the true cost of going faster

Day 49 ~75 minutes Field Day

Day 49 of 66

Field day: the two artifacts you'll actually be asked for

The last two concept days gave you calibration and the agreement diagnostic. Today is field work — you produce the two documents a lead is asked for in the first week of running a team: an onboarding checklist that brings a new annotator to live work safely, and a weekly throughput-vs-quality report that tells leadership the truth about the speed/quality tradeoff. Both are in your Week 10 portfolio.

Part 1 — The onboarding checklist

A new annotator who starts on the live queue cold is a noise generator pointed at your training set. Onboarding is the gate that converts a willing person into a calibrated one before a single live label counts. The non-negotiable element is the calibration gate from Day 46: live work is earned, not granted on day one.

Core Theory — the onboarding ramp
  1. Tools & access. Annotation tool, keyboard flow, where to ask questions, how to flag an ambiguous case (the Judgment Ritual, Day 5). Boring, but a confused tool is a quality problem.
  2. Guidelines, read and discussed. They read the guideline and walk the worked examples and edge cases with you. Reading alone produces confident misreadings.
  3. Gold practice. They label a set of known-answer items. You both see exactly where their reading diverges from the gold — low stakes, high signal.
  4. The calibration GATE. They label the team's calibration set and must clear a quality bar (e.g. ≥ 80% agreement-with-consensus on it) before any live item counts. Miss the bar → another round of practice, not the live queue.
  5. Ramped first week. Their early live labels are reviewed at a higher rate, throughput expectations are explicitly low, and you give the Day 46 feedback (specific, anchored, two-way) on the first batch.
The domain-expert track

Run the same checklist for a domain expert, with one branch: they still clear the calibration gate (their reading of your guideline isn't pre-calibrated just because their domain knowledge is), but when they miss the bar you investigate before assuming they're wrong — the expert's disagreement may be the guideline's error, per Day 47. Their gate is "calibrated to the guideline," and their disagreements feed guideline review, not auto-correction.

Part 2 — The throughput-vs-quality report

Leadership wants volume. Quality is invisible until it isn't. The single most valuable thing a Human Data Lead does upward is make the tradeoff visible and honest — to report both numbers side by side, every week, so nobody mistakes "we labeled 12,000 items" for "we produced 12,000 good labels."

What the weekly report shows
Honesty is the job

It is tempting to report only throughput when quality is soft, because throughput is the number that gets praised. Don't. A lead who hides a quality dip ships a worse model and owns the blame when it surfaces downstream — far more expensively, far later. Reporting the tradeoff straight, with a recommendation, is what makes you the person leadership trusts with the next decision.

The quality cost of a throughput push — quantify it

"Can we go 50% faster this week?" is a question you will be asked. The wrong answer is "no, quality will suffer" (unmeasured, easy to overrule). The right answer puts a number on the cost so leadership decides with eyes open.

Core Theory — modeling the cost

A throughput push usually comes from three levers, each with a measurable quality cost:

Put it in one sentence leadership can act on: "We can hit the deadline by dropping safety items to 1-way consensus — that removes the agreement check on ~30% of the batch and, from last month's audit, raises expected error on those items from 4% to roughly 9%. Your call; here's the number."

That sentence is the whole skill of this day. You didn't refuse, and you didn't quietly absorb a quality hit nobody chose. You converted a vibe ("quality will suffer") into a decision leadership can own. That is what reporting upward actually means.

Your work today

Produce both field artifacts

Onboarding checklist + weekly report

  1. Write the onboarding checklist for a task you know: tools, guidelines, gold practice, the calibration gate with an explicit pass bar, and a ramped first week. Add the domain-expert branch.
  2. Build a weekly throughput-vs-quality report template: throughput, gold accuracy, agreement, drift, and a one-line recommendation. Fill it with believable numbers for one week.
  3. Write the throughput-push sentence: pick one lever (consensus, gold rate, or per-item time), quantify the quality cost from real-ish numbers, and frame it as leadership's decision.
The lead move

An IC answers "can we go 50% faster?" with "no, quality will suffer" — unmeasured and easy to overrule. A lead quantifies the cost and hands the decision back: dropping to 1-way consensus removes the quality check on ~30% of the batch and raises expected error from 4% to ~9%. You converted a vibe into a decision leadership can own.

Say this in an interview: "I report throughput and quality side by side every week, including the dips — hiding a quality drop just ships a worse model and earns me the blame later. And no new annotator, expert included, touches the live queue until they clear a calibration gate."

Today's Takeaways